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EXECUTIVE SUMMARY 
PURPOSE 

Automated systems that support States in determining eligibility and calculating benefits for food 
stamp recipients can help States handle increasing caseloads more efficiently and cost effectively. 
The United States Department of Agriculture (USDA) Food and Nutrition Service (FNS) shares 
in the cost of developing State automated systems along with the Department of Health and 
Human Services (DHHS) when the systems are integrated to serve Aid to Families with 
Dependent Children (AFDC) Program and Medicaid recipients. As a principal provider of 
Federal funding, FNS is responsible for reviewing State requests for funding. If approval is 
granted, FNS also is responsible for overseeing the development and implementation in the State 
assuring the appropriate utilization of Federal expenditures, and providing assistance to States 
as needed. 

The purpose of this evaluation is to: 

• Review the effectiveness of the Advanced Planning Document (APD) process, 
identifying its strengths and weaknesses, and recommend improvements in the 
process. 

• Determine the staff time and level of ability necessary to evaluate APD requests 
in the regional offices (RO) and headquarters (HQ). 

• Determine the best method for applying standard system requirements to 
individual State system needs in the APD approval process. 

There were no study requirements to determine the cost for the current system of APD reviews 
and approvals, or estimates of the cost savings possible if the recommended improvements were 
implemented. There are, therefore, no dollar impact estimates as part of this evaluation. 

Background 

FNS personnel approve or recommend approval of Federal funding for the State based on the 
APD review. Since the automated certification and issuance systems also support other 
Federally-funded public assistance programs, such as AFDC and Medicaid, the State systems 
have become integrated to minimize the duplication of effort and to make the administration of 
these programs at the State levels more efficient and effective. The APD approval process also 
is required by DHHS for the funding of the AFDC and Medicaid Programs. States submit their 
APDs to both FNS and DHHS to request Federal funding from both agencies and both agencies 
must provide approval of the systems approach and feasibility prior to the State undertaking the 
effort. 

State automation has met with varying degrees of success. Although automation has made it 
possible for States to continue to serve the increasing welfare population, States have not been 
able to demonstrate decreases in operating costs, reductions in error rates, reductions in staffing 
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requirements, or increases in claims collections that could be attributed to the automated systems 
that support the program. The General Accounting Office (GAO), in a review of Federal 
oversight of these system development efforts, found the oversight ineffective, citing systems 
thai have failed and systems that have cost far more than nrijHnallv hudeeted and funded. 
Although FNS has taken a number of actions in recent years to improve the APD approval 
processes, FNS continues to seek ways to make its processes more effective and improve 
communications with other Federal funding agencies. This report reflects the findings of a study 
of FNS' APD approval process. 

Findings 

1) FNS Regional Office Organization/ APD Review Consistency 

Each RO has the flexibility to staff APD reviews in a manner that seems appropriate to 
its specific organization. This leads to inconsistencies in how the APD review process 
is conducted and can create an environment where each segment concentrates on one 
specific aspect of the APD, but no one has an overall view of the full scope of the 
document. 

2) Staffing 

The majority of the APD review coordination falls on the information resources 
management (IRM) staff with added burdens being carried by program and financial 
management (FM) personnel. The workload includes support of ongoing State systems, 
as well as APDs for the Food Stamp Program (FSP) and the Special Supplemental Food 
Program for Women, Infants, and Children (WIC). There are times when the workload 
precludes an adequate amount of time for timely review and response to APD 
submissions. Additional full-time staff or the use of external resources to support ROs 
or HQ, as needed, should be given serious consideration. 

While the emphasis of the system is aimed at providing cost-effective and thorough 
program support, the general design stage (program functionality) and user acceptance 
testing are the two specific aspects of a project where FSP functionality can be reviewed 
and evaluated. The APD review process lends itself to an overview of the application, 
and detail regarding staffing, project management, development tasks and cost. These 
areas are more aligned to the IRM with some cost aspects addressed by FM. The impact 
of insufficient staffing is felt more seriously in the IRM section due to the level of 
oversight and involvement in the project process. 

3) Technical Expertise/Training 

Due to the lack of a technical, hands-on systems environment within the region, technical 
staff do not have the opportunity to use their skills; therefore, these skills erode. 
Training courses have become victims of tight budgets, compounding the problems of 
finding a way to keep a technical staff technically current. More training time and direct 
exposure to State data processing environments need to be available to provide the IRM 
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staff with an adequate foundation of knowledge to effectively review technical APD 
plans. Additionally, program and FM staff should be given the opportunity to gain more 
technological insight through the use of introductory courses addressing technological 
capabilities, terminology, and acquisition techniques and alternatives. 

As an alternative, external resources or expertise to provide in-depth technical 
understanding and experience should be acquired. This type of resource could provide 
the depth of experience needed to analyze APDs without having to increase permanent 
staff in the ROs. 

4) State Visit Time 

One way to better understand a State's APD plan and keep in touch with changing 
technology is to spend as much time as possible in the State's technical and planning 
environment. Time and funds for State visits are far too limited to enable RO staff to 
spend even the minimum amount of time at the State facilities. Without this type of 
foundation to understand the hows and whys of an APD, the 60-day review cycle will 
not allow even a cursory working knowledge of what the APD plan may accomplish. 
With this type of direct State interaction and understanding of the APD content, the 
current 

60-day review window may even be reduced to 45 days. 

5) Documentation 

It was very difficult to find any RO process that would allow the staff to monitor the 
progress of a State's technical effort to install or enhance its public assistance system. 
Additionally, there was no mechanism to track the expenditure of funds against the 
planned expenditure in the APD to determine if the project was meeting planned 
objectives. In many cases, there were no records to determine what a State's final 
system development costs were. Improved tracking will enable FNS to identify problem 
areas earlier and initiate corrective action much sooner. 

6) States' Perspectives of the APD Process 

Generally speaking the States felt that: 

• APDs are difficult to create and FNS does not provide enough guidance and 
support. 

• The approval process takes too long and very little feedback is provided regarding 
shortcomings and areas where improvements could be made. 

• The process was valuable, but there were major changes that needed to be made. 

• The dual APD relationship with FNS and DHHS makes the process much more 
cumbersome, wastes time, and, at times, creates conflicting requirements. 
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• More information concerning APD content and guidelines for addressing 
acceptable limits for project costs and implementation timeframes would greatly 
benefit the APD creation and approval process. The current environment has 
created an atmosphere of isolation and lacks the cooperative attitude that would 
facilitate the approval process. 

Recommendations 

1) Staffing 

FNS should increase staffing levels and/or use external resources to enable RO staff to 
spend more time in the States to learn, first-hand, of the project plans, keep more 
technically current through close observation of technology and development standards, 
and become more of a partner with the State. While it is acknowledged that there are 
current constraints on staffing levels, it is our belief that without a substantial increase 
in staff size, FNS oversight will continue to struggle to provide cogent and technically- 
sound support for the States' automation efforts. The other alterative is to greatly reduce 
the FNS oversight role and allow the States to be more responsible for total planning and 
development effort using Federal dollars. 

2) Travel and Training Budgets 

To enable RO staff to provide more effective oversight for the automation projects, more 
time needs to be spent at the State facility to gather information on project planning, 
development activities, problem areas, and the State's ability to implement a solid, 
workable solution. For active projects, visits to the State should occur monthly; for 
States without any active projects, quarterly visits should be sufficient. Technical 
training should encompass at least one course annually for each person involved in 
reviewing APDs. As a result of spending more time at the State site, RO staff will have 
a much more detailed understanding of the capabilities and drawbacks of various 
technical solutions and provide more effective project oversight. 

3) APD Evaluations 

For every RO, FNS should create an APD evaluation team, chaired/staffed by a senior 
member of the RO with representation from FM, program, and IRM staff to review 
comments of RO staff who have spent time with the State developing the APD on the 
strengths and weaknesses of the document. The evaluation team will make a formal 
recommendation for acceptance or rejection of the plan. This type of review should help 
to provide a more consistent view of State APDs and enable FNS to enforce a common 
view of agency procedures and regulations. An intermediate level (threshold) of RO 
approvals, such as $10 million, could be established that would reduce the HQ Executive 
Oversight Committee review to only those projects in excess of the new threshold. 
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4) Project Tracking 

FNS should create a new process for monitoring the progress of technical goals, 
milestones, and zczi expenditures. The process could be as simple as a PERT chart 
depicting the technical milestones/checkpoints, activities to be completed at the 
checkpoint, and planned expenditures from project funds by that time. A review of 
technical achievements, explanations of delays encountered, contingency plans to correct 
the deficiencies, and project costs (to date) should be conducted. Any deviations from 
the technical or cost plans can then be reviewed to determine the magnitude of the issues 
and the ability of the State to correct the deficiencies within an acceptable timeframe. 
This type of process should enable FNS to more clearly identify problem areas earlier 
in the project process and help reduce risks of major project failures. 

5) State Assistance 

FNS should create an RO or HQ process that will provide information on all State 
project activities and enable States to request more timely and specific information on 
FNS requirements, and other States' APD activities, and content. This recommendation 
is based on nearly universal feedback from the States on the lack of definitive cost 
allocation guidance from FNS, as well as a lack of national clearinghouse of information 
on other State activity. New approaches on how to better provide specific cost allocation 
information and other APD-oriented information in the form of guidelines, procedures 
or an informational clearinghouse should be considered. 

6) DHHS/FNS APD Coordination 

FNS should create a new process for coordinating the responses to State APDs between 
DHHS and FNS. The States' biggest concern with the present process is the additional 
time and effort required to resolve differences between the two agencies. Use of joint 
FNS/DHHS HQ reviews to identify areas of mutual concern, or to coordinate efforts to 
resolve inter-agency differences within the projected 30-day window would greatly reduce 
the dissatisfaction levels currently being experienced by the States. 

7) Post-Implementation and Cost-Benefit Reviews 

FNS has dropped the requirement to conduct post-implementation reviews of State 
projects due to the shortage of resources. We believe that the process of revisiting the 
implemented system to verify its functionality, performance, and accuracy and to 
establish the actual benefits achieved through implementation of the system needs to be 
re-established and become more formalized. APD justification and cost-benefit 
projections are seldom reviewed after implementation to determine what was really 
achieved and the strengths and weaknesses of the particular project's development, 
conversion, training, and implementation plans usually are not evaluated. In the absence 
of a review. process, the ability to abstract important data to better plan the next major 
automation effort is lost. A review process should be implemented to address: 
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functionality, eligibility determination and benefit calculation accuracy, performance 
(e.g., response time, batch cycle time) acceptability, and cost-benefit results achieved. 
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I. INTRODUCTION 
Background 

The Food Stamp Program (FSP) is a major element of our nation's support of low-income 
households and individuals in need of assistance. The program, which is administered by Food 
and Nutrition Service (FNS), provides food stamp benefits through State, county, or other local 
agencies responsible for the local administration and daily operation of the program. Designated 
agencies are given the mission of establishing recipient eligibility, issuing benefits, and reporting 
to FSP officials. 

The complexity of data and decision processes associated with executing this mission, along with 
the massive caseloads carried by many State and local agencies, has encouraged States to 
develop automated data processing (ADP) systems. Because these systems are relied upon to 
determine individual and household eligibility and calculate grants and food stamp benefits, the 
systems must be dependable and accurate with respect to Federal and State regulations. The 
systems also must treat all applicants and recipients uniformly. 

The Federal government provides funding to States for the development and operation of 
automated systems that support Federal public assistance programs. For integrated systems, 
reimbursement is provided by the Department of Health and Human Services (DHHS), FNS, 
and other system users according to a pre-approved cost allocation plan. 

A State desiring to implement a Food Stamp Program certification system with Federal matching 
funds must first seek Federal approval for the initial automation planning and incurring costs 
associated with the planning process. Federal approval is sought in the form of an Advanced 
Planning Document (APD). During the planning phase, the State must justify the development 
or transfer of a system based on the estimated costs and benefits of the system and submit this 
along with a plan for allocating the development costs across participating public assistance 
programs. Federal reimbursement for development is intended to encourage States to automate 
program operations to reduce the ongoing operational costs associated with administering the 
Federal programs. FNS offered enhanced funding for some State planning activities as well as 
for development. 

Since the early 1980s, FNS has employed the APD approval process to fund and monitor State 
automation progress, but this process has not always resulted in the development of cost-effective 
and efficient State systems. To improve and strengthen FNS oversight and reduce the incidence 
of system failures, the following actions were taken: 

1. FNS required States to "investigate the possibility of transferring all or part of a 
system from another State unless it can be documented that the design and 
operation of an alternative system would be less cosdy." 1 



1 FNS Handbook 901, Number 91-36, April 7, 1992, Page 9. 
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2. FNS offered enhanced funding for State planning and development. 

3. FNS suggested that States employ a monitoring contractor to assist with 
contractor and project management oversight during system development anH 
implementation. 

4. FNS revised the APD Handbook in 1992 and distributed it to all States. 

FNS continues to approve and fund State system development efforts through this process, 
basing its decisions on the information, project plans and cost estimates presented by the States 
in APDs. Since developmental costs have continued to increase and operational costs have not 
been reduced but remain constant, FNS is re-examining the APD approval process to determine 
whether the process needs further improvement. 

FNS has noticed that ongoing operational costs for automated systems vary greatly from State 
to State as do the costs associated with system development efforts. In some States, system 
development efforts have taken longer than anticipated resulting in additional, unexpected costs. 
States requiring new systems are encouraged to transfer existing "successful" systems from other 
States unless they can determine that a new development effort is more beneficial. States are 
encouraged to transfer existing systems in order to reduce the timeframe for design and 
development and control costs; however, most States feel they must customize the transferred 
systems to meet their own needs and requirements, leading to additional time and funding. 

In a 1990 study, the General Accounting Office (GAO) reviewed the status of FSP automation 
in several States. The GAO report documented that FSP systems in all States were automated 
to a certain extent. Twenty-four States had Statewide automated systems that integrated FSP 
functions with those of Aid to Families with Dependent Children (AFDC) Program, administered 
by DHHS. The GAO report indicated that few States had been able to realize the benefits of 
automation originally anticipated. It also addressed the costs associated with development and 
operations as well as the effectiveness of automation. 

In the 1990 Farm Bill (Section 1763), Congress requested that FNS conduct operational reviews 
of State systems to determine: 

• State compliance with conditions of initial funding approvals. 

• Whether these State systems adequately support program delivery, as required. 

Congress also required that FNS review the regulations and standards for approving State 
automated data processing and information retrieval systems to determine whether, and to what 
extent, the regulations and standards contributed to a more efficient and cost-effective program. 
FNS also was required to report to Congress on the extent to which State agencies have 
developed and are operating effective systems that support Food Stamp Program delivery in 
compliance with the Food Stamp Act. 
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Purpose and Objectives of the Study 

The purpose of the State Automation Systems Study is to assess the extent to which States have 
ucvclvjp^ cm J axe operating efficient and effective deliver}' systems and FNS is able to provide 
oversight and approval of these systems. This study requires an evaluation of the levels of 
automation, system features, relative efficiency, and cost-effectiveness of State ADP systems. 

The primary objectives of the study are: 

1. Describe and assess the current degree and state of development of State 
automated data processing and information retrieval systems. 

2. Assess and evaluate FNS' ability to provide oversight and determine appropriate 
funding levels for the development of State automated systems through the APD 
approval process. 

3. Compare State system development processes with industry standards for 
technological development. 

4. Determine what level of automation is necessary, desirable, technically sound, 
efficient, and cost-effective for handling FSP caseloads and the needs of FSP. 

5. Provide guidelines for assessing the reasonableness of State funding requests for 
ADP development activity. 

6. Examine operational cost-accounting and cost-control measures and practices 
which are or should be built into the project planning process. 

7. Identify the factors influencing a State's ability to implement regulatory changes 
in a timely manner. 

8. Identify the factors influencing the success of technology transfers from one State 
to another. 

The information gathered is to be used to ascertain the level of effectiveness and efficiency that 
has been attained through the implementation of automated public assistance systems. An 
analysis of each State's planning and implementation efforts and the results achieved will be used 
to better assess what processes have been successful and which procedures have had less than 
favorable results. The appropriateness of transferring other public assistance systems, as the 
basis for a State's new automation efforts, will be reviewed based on the amount of time and 
effort required to customize the programs to meet each State's public assistance functional 
requirements. The ability of RO staff to monitor and assist the State effort to plan, develop, and 
implement an effective public assistance system will be reviewed, along with the ability of both 
State and RO staffs to utilize the APD process to oversee the design and implementation of an 
acceptable, cost-effective, and efficient system. 



THE ORKAND CORPORATION 



9 



Table of Contents 



The findings of this study will be used by FNS to reevaluate the current standards and 
procedures in use, review and approve the automation efforts of the States, and determine which 
technical solutions/approaches are most successful in this automation process. The results of this 
study will iuclude the development cf three items: 

• A description of current State systems and their ability to meet current and future 
program demands; this description will address current technological standards 
appropriate to FSP, the impact of anticipated future demands on FSP and 
automated functions at the State level, and future directions that should be taken 
to meet those demands. 

• An evaluation of Federal approval processes and guidelines for funding State 
automation; FNS can use these guidelines in proposing standards for State 
systems and approving State requests for funding. Recommendations for the 
preparation of technology transfer analyses will be a specific result of this 
evaluation. 

• Information and recommendations to FNS regarding the areas specified in the 
Farm Bill: (1) system transfers; (2) documenting the costs and benefits of 
automated systems; (3) FNS operational reviews; and (4) assessment of the 
effectiveness of State automation. 

Purpose and Objectives of This Task - Evaluation of the Federal APD Approval Process 

The specific purpose of this task within the overall State Automation Systems Study is to review 
the current APD approval and oversight process being used by FNS to identify those aspects of 
the process that are effective and should be continued and those aspects that are less effective 
and should be reworked or eliminated. 

The task was accomplished in three phases: 

• Each of the seven FNS ROs was visited and personnel actively involved in the 
APD review process were interviewed. The interviews dealt with the specific 
procedures in use at each office, levels of technical knowledge, interaction with 
State staff in dealing with automation projects, and office APD documentation. 

• FNS HQ staff involved in the Executive Oversight process were interviewed to 
gain -their perspective on the strengths and weaknesses of the current process. 

• Systems and program staff were interviewed during each State visit to discuss the 
benefits and drawbacks of the APD process for the States and obtain their views 
on their interactions with all Federal agencies involved in the approval and 
funding process. 

This report will address our findings based on the information gathered from all sources and 
recommendations to be considered to improve the APD review and approval process. 
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II. METHODOLOGY 






This section discusses how The Orkand Corporation approached the task of evaluating the 
effectiveness of the FNS APD process frerr. the persp*^ riw of hnth FNS and the States 
themselves. Areas that will be addressed in this section include: 




• Timing of the visits to FNS HQ, FNS ROs, and the States. 






• Types of information gathered during the visits. 







• State reaction to the new FNS 901 Handbook for APDs. 



Regional Office 

The seven (7) FNS ROs were visited prior to the conduct of the State visits. This was done to 
enable the two site visit teams to gather insights into the current automation status of each State 
and collect information on the FNS aspect of the APD process. Part of each RO visit consisted 
of a discussion, with State liaison staff, to learn as much as possible about the current 
automation posture of each State prior to the State visits. 

RO visits began in June 1992 and each visit lasted three days. The visits each consisted of four 
parts: 

• An initial presentation of the purpose of the State Automation Systems Systems 
Study and the RO review. 

• Interviews with representatives from the FM, program, and IRM areas to discuss 
roles and responsibilities. 

• Reviews of APD and State files. 

• An exit interview with RO management to discuss any outstanding issues with 
missing documentation or other information regarding the office APD review 
operation. No specific discussions were held on findings of the visit. 

The opening presentation/kickoff meeting was normally attended by the ranking management 
representative (usually the Regional Administrator), management staff from FM, and the 
program and the external IRM staff supporting State systems. The Orkand Corporation provided 
an overview of the State Automation Systems Systems Study which included the reasons for its 
undertaking, the goals and objectives of the study, and the process for conducting State visits. 
Questions were asked to determine the organization of the RO staff, APD responsibilities, issues 
pertaining to the APD process, and specific State situations within the region. A schedule for 
interviews and document reviews also was established at the opening meeting. 

In most cases, the interviews were conducted with staff personnel. The focus of the interviews 
with personnel from each of the three staff areas was to determine which area had the 
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responsibility for control of the APD process, what role each area had in the review of APDs, 
and how comments from the three areas were gathered and a consensus was created from the 
individual viewpoints. Staff were encouraged to discuss the strengths and weaknesses of the 
process and describe how the three areas interacted to reach consensus. Special attention was 
given to how comfortable both the financial management and program staff were with dealing 
with technical documents and plans and how much exposure and training they had received in 
preparation for reviewing these types of documents. Additional attention was given to the 
backgrounds, experience, and technical training of IRM staff and the opportunities for them to 
receive ongoing technical training updates. Most of the regions had some interaction with their 
DHHS counterparts, but it seemed to be more informal person-to-person contact, rather than a 
formal agency-to-agency arrangement. 

A review of State APDs and RO State files was conducted to determine the amount and quality 
of documentation maintained by the region. In addition, this review provided the site visit teams 
a chance to get first hand experience with APDs and to ascertain how well or poorly the States 
were following the APD guidelines. Correspondence files were reviewed to help determine the 
level of assistance requested by the States and the level of support being provided by RO staff. 
Additional documentation provided some insight into the level and quality of detail maintained 
by RO on status of State automation efforts and ongoing operations. Cost records were also 
inspected to help determine the effective the States were in meeting estimated project costs for 
automation efforts and any other data that would help determine the cost effectiveness of these 
solutions. 

FNS Headquarters 

Three interviews were held with FNS HQ staff to determine the roles and responsibilities of the 
FM, IRM, and program areas. In addition, Mr. Joseph Leo, FNS Deputy Administrator- 
Management, was interviewed to gain the perspective of a senior manager and member of the 
Executive Oversight Committee on the APD process. Emphasis was placed on the APD roles 
undertaken by the IRM, FM and program operations staff, the current problems with the APD 
process, and areas of concentration to be pursued during the State visits. 

In addition, State files from IRM and specific background information regarding State statistics, 
such as participation levels, error rates, and administrative costs, were reviewed or gathered 
during these interviews and visits. This information would be used in preparation for State visits 
and incorporated into the individual State reports. 

State Visits 

The visits to each State began in February 1993 and continued into December 1993. Two site 
visit teams, each consisting of three members with experience in FSP operations, cost accounting 
and allocation, and systems analysis, handled approximately 25 States each. Each visit lasted 
for three days and was conducted under the following format: 

• Use of pre-visit questionnaires requesting information requiring some time and 
effort for the State to assemble. 
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• Introduction/kickoff meeting. 

• Interviews with specific areas and management staff for FSP operations and 
systems. 

• FSP system demonstration and data center tour. 

• Review of State APD and correspondence files. 

• A sampling of FSP system users to ascertain their level of satisfaction with the 
current system. 

• An exit interview to identify any data elements that were still outstanding and 
whose responsibility it was to resolve them. 

Data Collection Instruments 

Nine different data collection instruments (DCI), which previously were approved by the Office 
of Management and Budget (OMB), were used to capture approximately 5,000 elements of 
information in each State. Three of the nine instruments addressed issues relating to the FNS- 
State relationship or the strengths and weaknesses of the current APD process. These were the 
Project Management Interview Guide, Management Information Systems Interview Guide, and 
the Food Stamp Program Interview Guide. Each of these instruments addressed specific aspects 
of either the interaction between the State and FNS for support and approval of the APD or 
experiences that the State had in working with APDs. Emphasis was placed on timeliness, level 
and quality of FNS support, positive and negative aspects of the APD process, the impact of 
dealing with both FNS and DHHS for APD approval, and areas of the APD process that could 
be improved. Based on input from FNS, specific examples of areas of improvement were not 
requested from the States during our visits. 

Program operations, State systems, and the State's project management staff provided their 
views. Project management staff had some of the most direct contact with FNS RO staff and 
the APD process. 

In addition to DCIs, the States were asked to provide APDs, APD updates (APDU), files of 
correspondence files between the State and FNS, and copies of the cost allocation plans 
submitted to FNS for approval. These were used to provide a chronological view of the APD 
approval process from the perspective of the types of changes or clarifications that were 
requested and the amount of time that elapsed in completing the process. 

Impact of FNS Handbook 

One of the goals of the State and RO visits was to determine the usefulness of the FNS 901 
Handbook in the preparation of APDs. The handbook was also used in reviewing how well ROs 
and States followed the APD preparation and approval procedures. The way ROs handled APDs 
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and the timeliness of their processing the APD was studied. At the State level, the helpfulness 
of the document in guiding the APD preparation was ascertained. 

Since the release ikic uf il»e FNS 901 handbook was April 1992, its use in the ROs and States ' 
was found to be very limited. Insufficient data was provided to allow useful analysis. Most 
States that followed a FNS handbook were using either the 103 or 151 version. No specific 
analysis was made of the conformity to the earlier version of the handbook. 

III. FINDINGS 

This report addresses the positive and negative aspects of the APD evaluation process from The 
Orkand Corporation's perspective of the FNS' RO role. It offers a composite view of the States' 
perspectives of the impact of the APD process on their efforts to develop systems to support 
public assistance operations and presents their view of the value of FNS' RO and HQ support 
of these efforts. j 

In the visits to the seven FNS ROs, data collection instruments were not used. Information was 
gathered through interviews with regional staff and reviews of available documentation. As a 
result, statistical data are not reported for the RO visits. Subjective evaluations, based on the 
APD evaluation requirements presented in the FNS Handbook 901 and RO operational 
observations, are reported. 

The State results were captured via OMB-approved DCIs and allow us to provide a 
representation of composite responses. The State findings were limited to their responses to 
DCIs which, in most cases, did not provide for any anecdotal feedback since this type of 
information was not requested by FNS during the creation and testing of DCIs. The information 
from the States will be presented in table formats with whatever additional information was 
captured. 

Regional Office Findings 

1) Regional APD Organization 

Each RO appears to organize itself as best fits its ability to handle the evaluation of 
APDs. Most have the process under control of the IRM staff, but one region has it 
under the program staff. There was also a diversity in RO roles that may lead to 
inconsistency among regions in the evaluation process since no two regions allocate staff 
and responsibility in exactly the same way. Based on the levels of cooperation within 
a RO and the personalities of the individuals assigned APD evaluation responsibilities, 
the current process can work well or with great difficulty. There is no way to measure 
effectiveness and capability of the effort among all ROs due to the differences in 
structure, composition, and control within each region. An effort needs to be made to 
align all ROs into the same organizational structure to enable better management review 
and controls from FNS HQ over APD evaluation performance. 
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In all cases, the APD evaluation workload is not the entire workload of the IRM staff. 
IRM staff is usually split between internal systems support (e.g., regional LANs, 
connectivity to FNS HQ) and external systems (e.g., APDs, State system issues). 
Fxugidja and financial n~.onagc~.cnt pcrccnr.el are normally assigned as needed to support 
a particular State when an APD is submitted. When an APD arrives, it is copied and 
distributed to each area for independent and specific review of its designated areas (e.g., 
cost allocation, system design, functionality). Comments then are collected, 
discrepancies are discussed, and necessary documentation created. Clarifications from 
the State are requested, if necessary, and the executive summary for the Executive 
Oversight Committee review is prepared for projects in excess of $1 million. It appears 
that this type of independent review can lead to each area concentrating on its specific 
concerns and missing the impact of the overall project. The use of a consistent RO APD 
review committee with representatives from each of the three functional areas (FM, 
program, and IRM) taking the technical lead for their particular expertise would 
eliminate the potential for misinterpretations and oversights caused by the independent 
review approach. This may entail the creation of an RO APD review team that will 
apply the same review criteria to every APD within the RO and create a more consistent 
view of APDs. Individual liaisons to the State from FM, program, and IRM would 
continue, and their input to the APD review would be of considerable importance to the 
insights behind the APD. 

Staffing 

There are normally two IRM staff, one program staff, and one financial management 
staff assigned to the APD evaluation process. The work is in addition to other 
responsibilities which take a great deal of time to perform. This results in the time 
available for an effective review of the very technical APD evaluations. 

IRM staff tend to have less seniority within the region and less systems experience than 
their counterparts in program and financial management sections have in their respective 
areas. In addition, since they get no direct technical reinforcement by having regular, 
hands-on exposure to the technology they evaluate, IRM staff find it very difficult to stay 
current with the strengths and weaknesses of the hardware/software platforms. Since 
they must share the APD workload for all of the States within the region, IRM staff can 
find themselves either inundated with APDs or involved with support of ongoing State 
system issues. Most regions have an active APD environment today and the staff is 
usually very busy. 

Since the IRM staff carries the highest technical evaluation burden, it would be practical 
to ensure that there are enough personnel on staff to effectively handle the APD volumes 
currently being generated by the States. Every region appeared to have fewer IRM staff 
than could be considered practical to support the ongoing State support workload and do 
a competent job in reviewing the APDs being generated for approval. 

Program operations normally assigns a single person to handle the APD evaluations for 
all States within the region. The individual spends a great deal of time reviewing system 
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functionality, leaving cost allocation and hardware/software design and configuration to 
the other areas. The individual's ability to use the APD document to understand the 
functionality of the proposed system is difficult, at best. 

Financial management normally will assign one staff member to review the cost aspect 
of an APD. The assignment may rotate among a number of individuals, based on 
workload or familiarity with the State's financial reporting. In one region, the financial 
management area has dedicated an individual to working with APD evaluations on a full- 
time basis. This approach seems to be the most workable solution to address the lack 
of consistency and timeliness that was observed in the handling of APD evaluations. 

3) Technical Expertise 

As mentioned earlier, the IRM staff has little, if any, opportunity to get first-hand 
technical experience while working as RO staff. Since the pace of technological change 
continues to gain momentum, the consequences for the IRM staff are that they cannot 
maintain technical currency without regular exposure to that technology in the field. 
Even those who have extensive systems backgrounds will lose touch within six to twelve 
months without consistent interaction with State or Federal technical staffs. 

For the most part, RO IRM staff felt technically out of touch and unable to spend 
adequate time with State systems staff to gain meaningful ongoing technical updates. 
With limited travel budgets available, time spent with the States is insufficient to keep 
RO IRM Staff backgrounds current. 

Neither program nor financial management staff who handled APDs seemed to have a 
reasonable grasp on the technical concepts being presented in APDs. One region has 
attempted to provide some exposure to technical concepts and terminology, but it has not 
been used by the majority of the offices. Since the usual content of the APD discusses 
technical items, this type of background and training seems to be necessary. 

4) Training 

Technical training for IRM staff, as well as training in concepts and terminology for 
program and FM staff, appears to be sorely lacking. All ROs felt that there were many 
opportunities to attend classes, seminars and symposiums, but there are never enough 
dollars available to take advantage of them. Again, this appears to be a case of missing 
an opportunity to provide career growth, enhance technical understanding, and enlighten 
the evaluators of new techniques and offerings that could help a State improve its design 
or otherwise enhance its system. These types of improvements in multi-million dollar 
systems could result in savings in the tens of thousands of dollars, much greater than the 
small training costs that each region would undertake each year. By taking advantage 
of these relatively inexpensive opportunities, FNS could improve RO ability to provide 
analytical oversight and give the program and FM areas more insight into the technical 
components of the State's solution. 
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State Visits 

Lack of adequate travel budgets prevents nearly all ROs from spending the necessary 
amount of time at the State to observe the ability o f smp tn nian anH dp.ve.1on 
workable automation systems to support FSP. Since each State invests major personnel 
and financial resources and contracts with external contractors in its projects, it seems 
reasonable and cost-effective for the Federal government spend sufficient time and money 
to insure that its investment is being well-spent. Additionally, there is no better way for 
the regional IRM representatives, as well as program and FM staff, to learn about the 
system, its capabilities, design characteristics, functionality, pilot and conversion plans, 
development/implementation schedules, and cost considerations than being on-site when 
these discussions are being held and decisions are being made. It seems logical to spend 
a relatively small amount of money to send RO staff to the States when hundreds of j 
millions of dollars are being invested in the products of these planning and 
implementation efforts. 

Every RO spent less than the desired amount of time in their States monitoring the 
progress of the automation efforts. In the larger western regions, it is more of a problem 
because longer distances to each State location result in many fewer trips than in the 
smaller eastern regions where all States can be visited by car. 

To expect that three or four people can review, assimilate, and comprehend a plan in 60 
days that a staff of 10 to 50 people have developed with 20 to 40 person-years of effort 
is unrealistic. It would be far better for FNS to have its staff oversee the development 
of APDs by sharing in the States' planning efforts and only using the formal APD as the 
vehicle to fund and contractually bind the Federal resources to a concrete plan that has 
already been reviewed and approved by FNS through its direct oversight of the creation 
of the document. With higher staffing levels at the RO, this could be accomplished with 
relative ease and create a much more productive and cost-effective oversight 
environment. 

It is impractical to attempt to provide a "target" staffing level for ROs since the APD 
review effort would vary based on the number of APDs in process at the same time, 
complexity of the project, and success record of the State in implementing previous 
projects. It would be more reasonable to establish what can be expected to be completed 
in the current 60-day window and how many projects can be handled concurrently. 
Based on these factors, staffing levels would be determined by the number of APDs to 
be reviewed and monitored. The number would vary based on the number of active 
APDS. 

Consistency of APD Process Across Regional Offices 

As mentioned earlier in the APD Organization section, each RO appears to have the 
flexibility to organize independently to evaluate APDs. One of the results of different 
APD review approaches is the lack of consistency among ROs on how APD rules are 
enforced. These inconsistencies included: 
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• Time when APDs are closed. 

• Conduct of post-implementation reviews (when they are required). 

• Conditions when APDs are required for hardware upgrades. 

• Guidelines for cost-allocation plans. 

• Timeliness in responding to APDs. 

We found that some ROs closed APDs after the approved system was implemented and 
some kept them opened and used APDUs to process requests for additional or new 
hardware and software or to perform some major enhancements to the system. 

In some cases, ROs performed post-implementation reviews within a reasonable period 
after the system had stabilized, but, in other cases, there was no evidence that a post- 
implementation review had ever been conducted, even on systems several years old. 

Some ROs required States seeking Federal financial participation to submit APDs for 
shared State data center hardware upgrades, even if there was no apparent connection 
with the FSP system. Others did not require APDs for these types of upgrades. 

Cost allocation plans appear to be viewed as independent entities, rather than integrated 
parts of APDs, and State staff believe that the lack of overall guidelines — regarding 
what was expected, what was viewed as acceptable, and what was needed to be reworked 
— complicated the preparation process. Even with the 1986 Memorandum of 
Understanding with DHHS as the recognized guideline, there were no clear rules or 
expectations that provided consistency among ROs. As discussed in the findings from 
the State visits, the States' biggest complaint regarding the APD process was the total 
lack of guidance and consistency in the cost allocation plan area. 

Timeliness in processing APDs within 60 days of receipt was interpreted differently 
across ROs. Some ROs felt that as long as you notified the States that the APD had been 
received and was under review that the requirement had been met. Others made every 
effort to get back to the State as soon as possible with comments, suggestions, and 
concerns. 

Without a common staffing approach to APD reviews in every RO and consistent 
application of the guidelines that were established in the APD Handbook 901, there may 
not be consistent, comparable APD reviews among the ROs. Each RO would follow 
its interpretation of the 901 Handbook, based on the makeup and perspective of the 
individual APD teams. Each APD review team would have different perspectives based 
on what area leads the APD study in a particular RO and which areas of expertise are 
represented on the review team. This can result in one RO finding a State's APD 
acceptable, while another RO may find a similar APD representation unacceptable. 
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Documentation 

As part of each HO review, time was spent reading documentation files on APDs. project 
histories, and State operating information to gain a better understanding of each State's 
condition prior to the State visit and to review how well the RO kept track of a project's 
progress. Expectations were that there would be an abundance of information on the 
APDs and the cost history of recent or active projects. 

What was found was a variety of conditions that ranged from full files of current and/or 
active APDs, including the APD itself and pertinent documentation, to files that had very 
sketchy background information and no APD documents. Part of the problem was tied 
to limited storage space, but it was clear that there were no specific filing requirements 
for State information and documentation. More importantly, we found it very difficult, 
if not impossible, to get any information as to the current cost history of any of the 
projects. Some ROs had information as to the amount of money being drawn by a State, 
but no other definitive data addressing how the project was progressing according to the 
FNS-approved development and implementation plan. There was virtually no 
information within financial management, program, or IRM areas that could show what 
had been authorized and what had been spent-to-date on any APD about which we 
inquired. The SF-269 report aggregates all development activity in a single column. 
States are not formally required to provide a detailed breakout of each development 
project and, without this breakout, there is no way to determine the cost history of a 
specific project. The same issue applied to operational costs, which are also combined 
into a single column on the SF-269. There appears to be no requirement for ROs to keep 
any detailed information on projects, even though it would be one of the most significant 
indicators for tracking progress of the project. 

Lack of this type of documentation and tracking data would seem to be a serious 
omission in FNS' mission to effectively oversee the State's ability to develop and 
implement an automation solution for FSP. 

Applying System Standards to State System Needs 

FNS should review the following areas to determine the effectiveness of the State's 
functionality and planning effort: 

• APD functional specifications and compare them to the Model Plan requirements 
for completeness. A new, more specific listing of Model Plan specifications may 
need to be created. 

• Inclusion of programmatic staff, field staff, in-house MIS staff in the 
requirements planning phase of the project process. 

• Inclusion of standard hardware and software characteristics for Request for 
Proposals (RFP). 
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• Reasonable performance requirement specifications (i. e. , response times, database 
sizes, transaction levels, etc.). 

* Project dcvclcpmcr.t, testing, conversion, and production rollout timeframes that 
fall within other States' experiences for States with similar workloads and 
functionality. 

State Findings and Perspectives of the APD Process 

FNS has experienced some problems with States not providing the required APD information; 
on the other hand, States have had difficulty, at times, in providing the information in the format 
required by FNS. Several sections of DCIs addressed the State's perceptions of the APD 
process and its value to the State. The questions were multiple choice to enable the responses 
to be more accurately coded. Open-ended questions were avoided to minimize the number of 
variables to be analyzed. This approach reduced the amount of information available as to the 
reasons for the States' perceptions and limits our ability to provide more depth than is depicted 
in the following tables. Information was collected on those areas that appeared to be most 
problematic to the States. Data collection focused on the kind of direction, guidance, or 
assistance that FNS provided to the State to minimize the number of APD iterations. 

The questions asked were approved by FNS and OMB for use in DCIs to gather States' 
perspectives of the Federal APD process. The approach focused on obtaining the States' view 
of the current process, rather than on obtaining specific recommendations and proposed changes 
from each State. This was done to establish an overall view of the process from the States' 
viewpoint and not to create a lengthy list of wishes specific to each State environment. The 
following tables reflect the composite attitude regarding the current strengths and weaknesses 
of the APD process. 

The opinions of State staff also were solicited about the helpfulness of the APD process and their 
suggested changes to the process. A review of this information should provide direction to FNS 
in areas where changes in the process could be made. Information regarding specific changes 
each State wanted to make was not requested because the focus of the collection was to quantify 
those areas where some type of improvements were desired. 

For those States that had direct knowledge of the FNS 901 Handbook, nearly everyone felt that 
it was a valuable document that would assist them in creating an APD. No State had, yet, used 
the 901 Handbook to prepare an APD. ROs all had the handbook, which was distributed in 
April 1992, for a period of three to eight months, but its procedures were not being followed 
in all regions as discussed in point #6 of the Regional Office findings. 

The following charts are a representation of those DCI questions addressing the APD process. 
The tables reflect the actual questions and possible answers that the State could select. Open- 
ended answers were eliminated to provide a more consistent pattern for analysis in the Final 
Report. They are being presented to give FNS a snapshot of the States' perspectives of the 
current process. 
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1) States' View of the APD Process 



The information presented in Table 3.1 pro"ides 3 nirhw r»f hnw well the States perceive 
the APD process has helped them. Although most of the States thought they had enough 
time to prepare their APD(s), most also felt that FNS did not provide a response to the 
APD in the expected timeframe. Most States could not differentiate the performance of 
RO and HQ as far as timeliness of the review process, but many felt that it took longer 
than anticipated. More than half had to request changes in their original request that 
resulted in additional Federal funding. 



Table 3.1 APD Review and Approval History 





Number of States 


Changes were required to original system design to stay within 
Federally-approved budget 


7 


Changes in the project required additional Federal financial participation 


27 


Had enough time to prepare APD(s) 


38 


Ever had an APD rejected by FNS 


13 


APDs have been processed by RO within 60 days: 




Usually 


15 


Sometimes 


5 


Rarely 


25 


None Submitted 


2 


APDs have been processed by FNS HQ within 30 days: 




Usually 


6 


Sometimes 


3 


Rarely 


22 


None Submitted /Unknown 


13 
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2) FNS Oversight 

The view of the State FSP personnel regarding RO oversight was difficult to capture. 
Responses regarding the effectiveness of FNS oversight are provided in Table 3.2. 
Many indicated that their project was too old to provide a representative answer or that 
they had not been required to complete an APD and were unable to provide any insight. 
The one item of note was the universal view that FNS does not provide enough specific 
information and guidance to enable the State to prepare an effective APD the first time. 
APD preparation is viewed as a "trial and error" process, especially in the area of cost 
allocation planning. 



Table 3.2 State FSP View of Regional Office Oversight Effectiveness 



Oversight Activity 


Number of States 


Harmful 


Ineffective 


Somewhat 
Effective 


Very 
Effective 


No 
Assistance 


Providing technical 
assistance in system area 





2 


6 





18 


Providing guidance in APD 
preparation 





2 


12 


4 


10 


Providing assistance in 
gaining APD approval 


3 


1 


15 


6 


5 


Assisting in understanding 
Federal Financial 
Participation requirements 





6 


9 


4 


10 


Assisting in correcting 
system deficiencies noted 
by FNS 





1 


5 


1 


17 
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Information gathered from MIS staff on their working relationships with FNS staff is 
quite complete (Table 3.3). More than half the States reported infrequent contact (less 
than monthly) with FNS staff. A majority of the staff who responded rated the FNS 
staffs capability and cooperation quite highly; however, 30 percent of the States that 
responded thought FNS policies and procedures were inconsistent. Over 47 percent of 
the States viewed FNS technical staff capability as poor or had no experience with FNS 
technical staff. This may indicate some dissatisfaction by State staff with the support 
provided by FNS, especially if States do not seek technical assistance from ROs because 
they perceive it as not valuable. 



Table 3.3 MIS Working Environment with FNS 





Number of 
States 


Frequency of contact with FNS staff: 




Monthly or more frequently 


10 


Quarterly 


15 


Less than quarterly 


22 


Capability of FNS technical staff: 




Very good 


11 


Good 


16 


Poor 


4 


No experience with technical staff 


20 


Cooperation between MIS and FNS staff: 




Very good 


14 


Good 


25 


Poor 


3 


No experience with FNS staff 


7 


Consistency/fairness of FNS policies and procedures: 




Very good 


4 


Good 


26 


Poor 


12 
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The relationships between the States and FNS during State development projects seem 
to have been satisfactory (Table 3.4). More than half of the States used some assistance 
from FNS, although many thought they could have used more assistance. More than 60 
percent of the States thought their technical and program relationships with FNS were 
satisfactory or better and many of them worked with FNS staff monthly or more 
frequently. This staff was part of the project management team and had more interaction 
during the project than the MIS staff who responded in Table 3.3. The States seem to 
have had less contact with any FNS financial management staff, as evidenced by the 
lower response numbers. 



Table 3.4 Relationship of State and FNS During Development Project 



Nature of Relationship 


Number of States 


Utilized assistance from FNS 


28 


Could have used more assistance from FNS 


31 


Worked with FNS technical staff monthly or more 
frequently 


18 


Technical relationship with FNS was 
satisfactory or good 


32 


Worked with FNS program staff monthly or more 
frequently 


27 


Program relationship with FNS was 
satisfactory or good 


35 


Worked with FNS financial management staff 
monthly or more frequently 


12 


Financial management relationship with FNS 
was satisfactory or good 


23 
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As shown in Table 3.5, less than half of the States have some contact with FNS on an 
ongoing basis. More frequent contact occurs between the State and RO than with FNS 
HQ. Just over half of the States reported on-site visits by RO staff during the project 
and an equal number reported telephone conversations with FNS HQ or RO staff. The 
audience for these questions was program staff who may not have been as directly 
involved as the system staff during the project. No specific information was requested 
as to direct interaction with FNS HQ staff or DHHS staff during State project 
development. 

Table 3.5 Ongoing FNS Oversight Activities 



Activity 


Number of States 


Post-implementation review conducted by RO 


22 


On-site visit by RO during system development 


26 


Telephone conversation with FNS RO 


24 


Telephone conversations with FNS HQ 


2 


No contact 


14 
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3) APD Process 

Most of the States' development project managers had an opinion about the helpfulness 
of the APD process for their States. Twenty-seven percent thought the process had been 
disruptive in their States (Table 3.6). Fewer MIS staff interviewed had opinions on FNS 
RO oversight and the opinions they had were mostly neutral. The smaller response 
probably reflects the lack of involvement by MIS staff in the APD process in many 
States. In general, the APD process was considered helpful in forcing the State to plan 
the project more effectively. The negative aspects were focused primarily on the 
approval delays and disruption to the flow of the project due to FNS requests for 
additional APD clarifications and APD rework without any compensating benefits that 
could be quantified (e.g., better implementation plan, more equitable cost allocation 
method). 



Table 3.6 Assessment of the APD Process by the State 





Number of States 


Rating 


View of State's APD Process 
by State's Project Manager 


View of FNS Regional APD Oversight 
by State's MIS Representative 


Helpful 


22 


7 


Neutral 


11 


23 


Disruptive 


12 


7 
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Various aspects of the development project were perceived to be affected to different 
degrees by the APD process (Table 3.7). Two project areas, cost allocation and project 
schedule, emerge as having been significantly affected by the APD process. 



Table 3.7 Impact of the APD Process on Various Areas of the Development Project 



Development Project Area 


Number of States Perceiving the Following Level of Impact 


None 


Little 


Some 


Much 


Budget 


14 


5 


8 


11 


Cost Allocation 


7 


4 


2 


29 


Project Cost 


13 


7 


10 


8 


Cost Reporting 


14 


5 


6 


14 


Project Scope 


23 


5 


4 


8 


Project Complexity 


29 


8 


1 


3 


Project Management Approach 


29 


6 


3 


4 


Procurement Approach 


15 


5 


1 


12 


Project Schedule 


8 


4 


7 


24 


Use of FNS Guidance/ Assistance 


8 


14 


2 


9 


Use of DHHS Guidance/ Assistance 


7 


10 


8 


7 
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Many States found several aspects of the APD process valuable in undertaking any major 
project. The value was in the planning and review of critical components of the project 
to ensure that all aspects had been considered effectively and contingency plans had been 
created. The APD process areas found to be valuable by most States are the planning 
and implementation APDs, the cost allocation and cost-benefit analyses requirements, and 
DHHS certification. 



Table 3.8 APD Areas with Value for States 



Areas of the APD Process 


Number of States that Value 
APD Process Area 


Planning APD 


27 


Planning Request for Proposal (RFP) 
approval 


6 


Cost Allocation 


21 


Cost Benefit 


29 


Implementation APD 


30 


Implementation RFP approval 


10 


FNS approval 


22 


DHHS approval 


23 


Certification 


31 


System transfer emphasis 


19 
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Many States also found several aspects of the APD process that they would like to see 
changed (Table 3.9). Both FNS and DHHS approval were selected by more than half 
the States as areas needing revision to be more effective for their needs. In addition, the 
cost allocation process generally was viewed as necessary, but a majority of the States 
felt that the current process created issues that were difficult and time-consuming to 
resolve. The primary concern revolved around the lack of definitive guidelines from 
Federal agencies as to what was acceptable and what was not acceptable in cost allocation 
plans. Several States felt that they were playing "guess the right number" when 
submitting financial planning data to both Federal agencies. Specific information was 
not gathered from the States about the types of changes they would like because such 
information was not requested by FNS. 



Table 3.9 Changes Regarding the APD Process 



Areas of the APD Process 


Number of States that 
Want Changes Made 


Planning APD 


13 


Planning RFP approval 


19 


Cost Allocation 


28 


Cost Benefit 


20 


Implementation APD 


16 


Implementation RFP approval 


25 


FNS approval 


30 


DHHS approval 


30 


Certification 


15 


System transfer emphasis 


22 
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A majority of the States indicated that the FNS and DHHS approval processes required 
changes. The specific improvements suggested in Table 3.10 provide a similar picture. 
The most popular suggestions are for FNS and DHHS to have more consistent 
requirements and for the APD process to be simplified. The States are frustrated with 
the process which requires them to conduct two simultaneous negotiations with FNS and 
DHHS for approval of a single APD. Many times the States are required to become a 
facilitator between the agencies to resolve different, and sometimes conflicting, 
modification requests. States perceive that there is very little productive communication 
between FNS and DHHS and the approval process would not be completed in any sort 
of timely manner without their proactive participation and bargaining. It is also clear 
from the responses that the States would like more assistance from FNS during all 
aspects of the project. Again, information regarding the status and approaches from 
other States' development efforts would be greatly appreciated. 



Table 3.10 Improvements to the Federal Government's Funding Approval 
and System Development Monitoring Process 



Improvements Suggested 


Number of States 


More technical assistance 


22 


More program assistance 


16 


More candidate system assistance 


31 


More consistency with DHHS requirements 


45 
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The nature of the dual relationships of the State with both FNS and DHHS is not 
perceived as being as cooperative or coordinated as they could be (Table 3.11). More 
than half of the States thought that having to work with both agencies was non- 
cooperative and time-consuming. Most of the States thought the dual role of DHHS and 
FNS complicated the APD review process, delayed APD approval, and resulted in 
requests for conflicting changes. Very few thought there were any positive effects of the 
joint DHHS/FNS APD effort. Again, FNS did not request that specific examples be 
captured from States during the site visit;' therefore, the data collection effort focused on 
the States' perceptions of the process. 



Table 3.11 Interaction Between FNS and DHHS 



Nature of Interaction 


Number of States 


Similarity of requirements: 




Similar requirements regarding system development 


9 


Different requirements, but efforts involved overlap 


28 


Completely separate efforts required 


7 


Degree of Cooperation: 




Cooperative and non-disruptive interaction 


2 


Non-cooperative and non-disruptive interaction 


11 


Non-cooperative interaction, creating a difficult, time- 




consuming environment in which the State must operate 


28 


Specific effects of dual role of DHHS and FNS in APD review: 




Complicated the review process 


38 


Delayed approval 


33 


Requested conflicting changes 


27 


Simplified APD approval 


1 


Provided complementary assistance 


6 
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CONCLUSIONS 
Organization 

There is a lack of consistency among ROs when dealing with APD reviews. The 
differences can be found in several areas. For example, for upgrades to peripheral 
equipment in a shared State data center, several ROs require that the State submit an 
APD or APDU, as required by the 901 Handbook. Other ROs allow the State to process 
the upgrade request within normal State acquisition channels. This may reflect the 
absence of a formal and recognized FNS organizational plan that could be used by all 
seven offices and could help create similar procedures and processes in all ROs in 
evaluating APDs. Under current procedures, different rules are applied to deal with the 
same issues depending on the FNS region in which a State resides. Since States 
communicate frequently when evaluating systems for transfer, it is reasonable to assume 
that they also would discuss APD reviews and issues raised by FNS during the approval 
process. If different issues and approval methods are used for the State receiving the 
transferred system, FNS presents the appearance of being inconsistent in the application 
of its rules and guidelines. 

Staffing 

Given the scope of the RO responsibilities -- handling both FSP and WIC APDs, dealing 
with other aspects of the States' ongoing operational systems, and attempting to remain 
technically current ~ there is insufficient IRM staff to support the workload. The APD 
review process lends itself to an overview of the application and detail regarding staffing, 
project management, development tasks and cost. These areas are more aligned to IRM 
with some cost aspects addressed by FM. While the emphasis of the system is aimed at 
providing cost-effective and thorough program support, the general design stage (program 
functionality) and user acceptance testing are the two specific aspects of a project where 
FSP functionality can be reviewed and evaluated. It would be more practical to 
determine RO staffing levels based on the level and complexity of ongoing State support 
activities in the RO and the expected level of APD review efforts. For example, because 
of the size and complexity of California's efforts and the project histories and automation 
needs in both Oregon and Washington, the Western RO would require more IRM staff 
than another RO would need to cover a similar number of States and projects. The 
attempt to staff ROs for an average workload seems to prevent FNS, at times, from 
having adequate staff and staff time to effectively address multi-million dollar project 
reviews. 

Each region should have staffing levels reviewed at least every two years to insure that 
there is a reasonable workload spread for support of State systems. 

State Visits 

Along with staffing, it is clear that insufficient time is being spent at the State facilities 
to observe and understand the activities and levels of performance being achieved by the 
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State. There is no better way to learn and understand how a project has been conceived 
and planned than by watching it develop and grow. In addition, RO staff can acquire 
better technical knowledge by working closely with State technical staff as they develop 
the plans and requirements for a new system than by merely observing the results that 
are achieved. On-site involvement of RO staff also helps build a sense of partnership 
between FNS and the State. This type of on-site activity also may provide FNS with 
first-hand knowledge of the project's progress and allow FNS the opportunity to identify 
negative trends and warning signs in the project cycle much sooner. This information 
may help FNS assist the State in creating solutions that do not significantly affect the 
overall success and cost of the project. 

During the planning and development cycle of an automation project, the RO 
representative should be on site at the State at least monthly and more frequently if the 
project is approaching a critical phase or shows signs of project slippage, cost overruns, 
or performance problems. All seven ROs lack sufficient travel budgets and staff time 
to spend this amount of time at a State to observe the project plan and staff in action. 
While it is important to control administrative costs, cost considerations must be balanced 
against the need to provide adequate oversight and technical support. Given that 
cumulative costs for State systems projects are hundreds of millions of dollars, the 
potential for significant cost savings through improved oversight justifies nominal 
increases in investment for staff and travel. Information gathered during the State visits 
indicated that 24 States would spend $943.2 million over a six-year period beginning in 
1990 to develop Federally-financed automated systems that would support FSP. An 
investment in two additional staff members with annual salaries of $40,000 in each 
region, two additional HQ staff with $40,000 per year salaries, and 40 percent for fringe 
benefits would cost FNS less than $6 million over the same six-year period. If improved 
oversight saved 0.7 percent of the estimated project costs, the costs for the added staff 
would be recovered. 

Assistance to States 

States are looking for FNS to participate as a partner in their automation projects, not 
as an implementor of the project plan. It is not FNS' mission to perform the project 
tasks, but it is FNS' role to provide assistance in program operations, financial 
management, and technical solutions as required by the State. State personnel feel that 
the quality and quantity of support being provided by FNS is insufficient to help States 
create effective APDs. This shortcoming creates delays in the creation and approval of 
an APD and consequently delays the development cycle of the project. Delays 
subsequently increase the cost of the project and FNS participation levels. Additionally, 
the desired partnership between the State and FNS will be negatively affected by the 
perceived lack of FNS interest or concern in the State's desire to improve its operational 
environment supporting FSP. 
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5) Training 

There is virtually no technical training being conducted for program, FM and IRM staff 
in the ROs. Lack of money and time for training prevents staff from getting necessary 
technical updates. If IRM staff do not deal with hands-on technical tasks due to the 
nature of their work, are unable to spend any amount of productive time in a hands-on 
technical environment (i.e., State IRM organizations), and lack training and exposure to 
new technological innovations, they will necessarily be unable to understand the issues 
and effectively review proposals for multi-year, multi-million dollar developmental 
efforts. The program and FM staff also are handicapped by their lack of familiarity with 
current hardware and software tools and techniques. Training would provide an 
understanding of the basic concepts and terminology, which would enable both the 
program and FM areas to be more productive in their reviews of the total APD 
document. Without a commitment to keeping RO and HQ staff abreast of the 
environment in which States operate, FNS has little chance to have any technical impact 
on the State's projects or any meaningful insight into the reasonableness of the State's 
implementation plans. 

6) Documentation 

Lack of APD documentation and history at ROs means that there is no effective audit 
trail for the State's conduct of the project. This makes it very difficult to establish 
review points, perform project tracking, and effectively report on project progress 
without relying on the State for accurate information. This was especially true in the 
cost areas where it was very difficult to find any meaningful data covering the project 
history in any RO. Detailed files should be maintained during the active life of a project 
and a summary file addressing those items identified as requirements should be 
maintained at ROs on an efficient storage medium for the life of the system. 

7) Tracking 

There was very little evidence that ROs were able to determine where a specific project 
was in its development or implementation cycle. Financial management had some 
separate reports that allowed them to determine how much money had been drawn down, 
but there was no way to assess if the project was on schedule in meeting performance 
and cost objectives established in the APD project plan. 

8) APD Review Consistency and Coordinated Requirements 

Each RO had its own method of allocating staff for APD reviews, dividing the APD for 
review and comment, and consolidating the results in a single response to the State. ROs 
also were not consistent in their definitions of timeliness in meeting the 60-day window 
for responding to APDs. In addition, there were very few instances where APDs were 
rejected by FNS as insufficient. There was almost always a conditional approval 
granted, sometimes after the APD had been in the process of evaluation for as long as 
a year after the initial submission by the State. It would seem to be more practical for 
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all parties if the approach were modified. All APDs would be reviewed within the 60- 
day window. For APDs that RO staff found to be significantly complete and accurate, 
FNS would grant conditional approval and allow the State to begin work while the final 
details were eiaiiucu. If uic uuvuinciii was found to be significantly incomplete and 
unacceptable, it should be rejected and specific reasons for its rejection would be 
included. The APD would be sent back to the State for a total rework. This process 
would place both ROs and States in positions of accepting responsibility for the quality 
and acceptability of APDs and would enable each side to concentrate on workable 
documents and clear requirements. 

V. RECOMMENDATIONS 

Based on the information contained in Section lit, Findings, and Section IV, Conclusions, we 
recommend the following actions to improve the FNS oversight and APD evaluation process: 

1) Staffing 

FNS should increase HQ and RO IRM staffing levels to allow for reasonable workload 
allocation between APDs and other State program activities. The level would vary by 
RO and the workloads currently in place. Staffing should be reviewed every two years 
to insure that the workload status justifies the staff level. Levels should be sufficient to 
allow for timely and regular State visits and for reasonable levels of ongoing technical 
education. Based on our observations and the voluminous amount of technical detail in 
APDs, it is recommended that no RO staff person handle more than two major (in excess 
of $10 million) APDs at the same time. Each staff member should handle no more than 
three States for all activities to help ensure that full attention can be given to each State's 
developmental effort. Additional support could be supplied through the use of qualified 
APD-experienced contractors. 

HQ staff should also be increased to allow for dedicated coverage to an entire group of 
States, delineated by RO. This would help create a more consistent delivery of support 
to ROs and their States. 

An alternative to increasing RO IRM staffing levels would be to use external contractors, 
exclusively, to perform the detailed APD evaluations and provide a summary report to 
the RO for its comments and concurrence. This would free RO staff to deal with non- 
APD issues, monitoring of implementation activities, or ongoing State system activities. 
The cost for this approach would be dependent on the volume and complexity of APDs 
to be reviewed. Based on a three-person team with program, systems and cost allocation 
experience, costs could range from $3,000 to $6,000 per week. 

2) State Visits 

FNS should provide adequate staffing and travel funding to allow RO staff to visit State 
sites at least monthly to observe and monitor State project activities affecting FSP 
operations, including WIC and Electronic Benefits Transfer (EBT). 
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3) APD Evaluation 

At each RO, create a team consisting of representatives from the program, financial 
management, and LRM areas and chaired by one of the senior members of the RO 
management team. The committee would review comments from the individuals 
responsible for supporting the State who have reviewed and commented on the APD 
submission. The APD evaluation team would make the formal recommendation ~ to 
accept, conditionally accept, or reject the APD - either to the HQ Executive Oversight 
Committee or the State itself depending on the APD amount. The primary roles of the 
RO APD evaluation team would be to ensure timeliness of FNS processing and 
consistency in the application of FNS rules and requirements as well as to expedite and 
simplify the approval process. 

FNS should create an intermediate approval of $10 million that can be dealt with by the 
RO APD evaluation team without HQ oversight. This should enable most hardware 
upgrades and small system enhancements to avoid the higher-level review steps. Larger 
hardware and software APDs will still follow the Executive Oversight process. 

4) Project Tracking 

A milestone and cost expenditure checklist should be a required as part of each State's 
APD submission. This list would be retained and used by ROs to track the technical 
progress in providing deliverables and documentation and actual dollars expended by 
milestone. This capability would enable FNS to determine more quickly if the project 
is staying on schedule, based on the performance and costs at each milestone. If the 
milestones provided are too infrequent, FNS should work with the State to create interim 
milestones to track progress in shorter increments. Checkpoints should occur at least 
semi-annually or, preferably, quarterly. Any variance from the plan should be reported 
by the State in a formal document and corrective action plans should be submitted. FNS 
should be responsible for understanding the reasons for and adequacy of the report and 
should increase its monitoring activity if necessary. 

5) State Assistance 

FNS should provide more assistance to the States concerning the preparation of APDs. 
States expressed a need to receive more support from FNS in areas concerning cost 
allocation methodology and plans, especially in determining acceptable levels of system 
cost and cost allocation approaches that were acceptable to both FNS and DHHS. 
Information regarding technical issues and programmatic issues were felt not to be as 
important, but FNS should improve its information dissemination process. 

One method would be to create an APD hotline at FNS HQ to answer both State and RO 
questions regarding APD preparation and content. Another would be to create a 
clearinghouse of State system information that would compile data on each State's 
experiences with transfers, conversion, implementation, hardware/software platforms, 
etc. and be able to respond to all State inquiries concerning these activities. A third 
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would be to establish a pre-APD meeting with the State to review the APD format and 
requirements and answer any questions that the State might have. This step would enable 
FNS to provide the most current information on APD expectations and would enable the 
State to get dear, concise data as to how to create its ATD. 

6) Improved Coordination with DHHS on APD Reviews 

Every State claimed that the efforts to meet requirements from both FNS and DHHS 
created a major obstacle in the APD process. The efforts, usually coordinated by the 
State, not the Federal agencies, resulted in confusion, contradiction, and increased costs 
due to project start delays and changes. It is strongly recommended that FNS take an 
active role in attempting to consolidate the process into a single step that would satisfy 
both agencies and eliminate undue delays for the States. Although this type of activity 
has previously failed, the coordination issue should not be ignored because the lack of 
coordination is a major obstacle to making the APD process an effective and cost-saving 
tool. 

7) Post Implementation Review 

The FNS post-implementation review requirement has been eliminated. FNS should 
reinstate and formalize the post-implementation review process, making it mandatory and 
providing a formal structure of tests, reviews, and expectations. This would enable the 
State to more accurately design a system to meet the FNS (Model Plan) requirements. 
It also would create an expectation that the system will be required to perform to meet 
design specifications and be measured against these criteria. The review also would 
provide a vehicle for FNS to close out the APD as complete. 

8) APD Cost/Benefit Review 

As part of the post-implementation review or as a separate step, FNS should require the 
State to formally review and document actual benefits, tangible and intangible, that were 
achieved with the implemented system. This review should include estimated costs and 
savings, actual costs and savings realized, explanations of variances experienced, and 
conclusions drawn from the results. This process would force the States to better 
determine and document their baseline situations and more effectively track savings 
achieved. It also would provide valuable data about savings potential for future projects 
undertaken by FNS and the States. 
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